How Rubrics That Measure Outcomes Can Complete the Assessment Loop

Program assessment of student learning includes the following steps: 1) involving all constituents to establish program goals, 2) developing measurable student learning outcomes for each of the goals, 3) developing measurable outcomes for each course that map to the student learning outcomes, 4) determining appropriate assessment methods in the courses, 5) creating assessment instruments (or rubrics) for each of the methods, 6) establishing benchmarks, 7) analyzing the data, and 8) using the results to improve student learning. This paper focuses on the last four steps by beginning with a generalized assessment plan for an undergraduate computer science program. A generalized rubric for computer programs is presented that measures selected student learning outcomes. This paper demonstrates how to apply the generalized rubric to two specific computer programming assignments. Benchmarks associated with the rubrics are suggested. Sample results are analyzed to identify problems and propose solutions—“closing the loop.” INTRODUCTION A major reason for developing an assessment plan is to determine how well a program is doing in meeting the goals faculty set for student success. The driving force in determining program goals should be the competencies the faculty want their students to exhibit and the skills they want their students to possess after completing the program. Since reasons for developing an assessment plan vary, e. g., to satisfy regional accrediting bodies or to obtain discipline-specific certification, it is not possible to adopt a single blueprint that will be appropriate for all. However, we can identify a few basic components of effective assessment plans [3, 5, 8, 15]. ABET has adopted specific definitions for the following terms: Program Educational Objectives and Program Outcomes [6]. Educational objectives are broad statements that “describe the career and professional accomplishments that the program is preparing graduates to achieve” whereas outcomes are measurable statements that “describe what students are expected to know and be able to do by the time of graduation.” Differences between the two were exemplified in 2009 [13]. This paper adopts the model: program goals and student learning outcomes both articulate that which we want our students to achieve by the time they graduate. Once the goals and student learning outcomes have been articulated, measurable course outcomes must be developed that map to the student learning outcomes to ensure that all courses in the program of study are addressing the overall student learning outcomes. This process also verifies that all student learning outcomes are addressed in at least one course. (In the remainder of this paper, the term outcomes will refer to student learning outcomes.) The next step would be to determine one or more assessment methods to measure the outcomes. A combination of both indirect and direct methods of assessment is desirable. Indirect methods include: student or faculty surveys, reflection papers, focus groups, and exit interviews where students are asked to discuss what they have learned and its impact on their learning. Direct assessment methods include programming assignments, projects, in-class tests, portfolios, oral presentations and the ETS’s Major Field Test [4, 11, 12, 16]. These methods require students to demonstrate what they have learned rather than to discuss what they have learned. In addition, for each of the methods deployed, scoring guides (rubrics) based on a set of performance criteria must be developed to help evaluate student work [9, 10]. At this point, faculty will want to establish thresholds (benchmarks) and develop processes for analyzing and reporting the data to the stakeholders. Data that is collected must be analyzed to identify problems, solutions must be proposed and implemented, and the process must then begin again—“closing the loop.” Problems must be tracked until they are resolved [7]. GENERAL CS GOALS & OUTCOMES A common starting point for creating goals and outcomes is to draw upon the institution’s mission statement to construct a set of program goals that articulate what the program hopes to accomplish. These should be high level statements that are then broken down into more specific student learning outcomes (SLOs). It is essential that these outcomes be explicit and measurable and that they target a specific skill level, such as those articulated in Bloom’s Taxonomy [14]. Here we present a set of program goals and student learning outcomes developed by the authors and other colleagues by combining and refining the goals and outcomes at their respective institutions. Goal I: Critical Thinking and Problem Solving: Students will develop problem-solving and critical thinking skills and use these skills to solve abstract and complex computing problems. Student Learning Outcomes Students will: A. develop abstract models and design a solution to a computing problem B. design an algorithmic solution using decomposition and stepwise refinement C. develop and design software solutions using different design methodologies, data structures and programming languages Goal II: Theoretical Foundations: Students will acquire a working knowledge of the theoretical foundations of computer science. Student Learning Outcomes Students will: A. use mathematical underpinnings of the discipline of computer science B. examine the correctness and efficiency of the design of a software system C. analyze the complexity and computability of algorithmic solutions Goal III: Ethical Responsibilities: Students will become informed and educated citizens in terms of their professional responsibility to address the social and ethical implications of the use of technology. Student Learning Outcomes Students will: A. recognize the ethical, legal and social implications of computing B. analyze the impact computing has on the global society C. ensure the security, privacy and integrity of data Goal IV: Communication and Interpersonal Skills: Students will acquire communication and interpersonal skills necessary to perform effectively in a technical environment. Student Learning Outcomes Students will: A. use oral and written communication skills to convey technical information effectively and accurately B. use their interpersonal skills when working in a team environment C. use interpersonal skills when working with those outside of computing Goal V: Professional Responsibilities: Students will be provided with a foundation for continuing education and growth in the field of computing. Student Learning Outcomes Students will: A. recognize the need for continued professional and educational development B. be prepared for graduate study or a professional career in computing. Skill levels for the outcomes would vary amongst programs thus reflecting the individuality of the institution. The learning outcomes articulated above can be mapped to ABET’s Program Outcomes [2]. The mapping in Table 1 relates ABET’s new criteria for 2009 [6] to our SLOs. ABET’s outcomes are separated into outcomes for all of computing (a through i) and additional outcomes for Computer Science (j and k). Table 1: ABET Mapping ABET’s Program Outcomes for CS Our SLOs (a) An ability to apply knowledge of computing and mathematics appropriate to the discipline II.A (b) An ability to analyze a problem, and identify and define the computing requirements appropriate to its solution I.B (c) An ability to design, implement, and evaluate a computer-based system, process, component, or program to meet desired needs I.A, I.C, II.C (d) An ability to function effectively on teams to accomplish a common goal IV.B (e) An understanding of professional, ethical, legal, security and social issues and responsibilities III.A, III.C (f) An ability to communicate effectively with a range of audiences IV.A, IV.B, IV.C (g) An ability to analyze the local and global impact of computing on individuals, organizations, and society III.B (h) Recognition of the need for and an ability to engage in continuing professional development V.A (i) An ability to use current techniques, skills, and tools necessary for computing practice V.B (j) An ability to apply mathematical foundations, algorithmic principles, and computer science theory in the modeling and design of computer-based systems in a way that demonstrates comprehension of the tradeoffs II.B, II.C involved in design choices[CS] (k) An ability to apply design and development principles in the construction of software systems of varying complexity [CS] I.C PROGRAMMING RUBRICS Since computer programming is a central part of the computing discipline, computer programming assignments are often used to measure whether one or more SLOs are being met. Course outcomes in programming courses would typically map to SLOs associated with Goal I, Critical Thinking and Problem Solving and some or all of the SLOs associated with Goal IV, Communication and Interpersonal Skills. For instance, written communication skills could be assessed in the readability of the written code and in the interaction with the user. Interpersonal skills with those outside of computing could be assessed if software is written for general use. Interpersonal skills in a team environment could be assessed for team programming projects. More specifically, SLOs associated with Goal II, Theoretical Foundations, could be assessed for a given program that simulates finite automata, or compares memory management schemes or develops a heuristic for a Hamiltonian circuit (Outcomes A, B, and C, respectively) and SLOs associated with Goal III, Ethical Responsibilities, could be measured by a programming assignment which implements encryption algorithms. The first step in developing a generalized programming rubric would be to identify the specific SLOs that are to be measured. The selections here are SLO I.B (algorithmic solution), I.C (program design), IV.A (user interface and code readability). The next step would be to decide on how many levels of performance is appropriate. Too few levels can make it difficult to fit a specific situation into a performance level and too many levels can make it difficult to develop meaningful differences among performance levels. The generalized rubric uses four levels: Unacceptable, Poor, Good, and Excellent. The last step is to describe performance at each level. One strategy might be to begin by describing the highest level of performance for the outcome and then back off slightly for each lower level. Scoring a rubric involves assigning a value or range of values to each of the levels of performance. For example, Unacceptable: 1, Poor: 2, Good: 3 and Excellent: 4. For a finer granularity, one could use a scale such as: Unacceptable: 0-3, Poor: 4-6, Good: 7-8, and Excellent: 9-10. Table 2 contains the generalized programming rubric. Table 2. General Programming Rubric Unacceptable Poor Good Excellent